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(57) Abstract 

A televisicMi audience measurement system measures viewing of a television program viewed on digital television located in a 
statistically selected site by (i) retrieving an audience measurement data packet fn»n a television set in oider to identify the television 
program, (ii) detecting an audio code embedded in the television program in order to identify the television program, (iii) extracting an 
audio signature from die television program in order to identify the television program, (iv) identifying the televisira program through use 
of a^ftware agent, and (v) selecting at least one of die retrieving means, die detecting means, the extracting means, and die software acent 
m order to identify the television program. 
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(57) Abstract 

A television audience measurement system measures viewing of a television program viewed on digital television located in a 
statistically selected site by (i) retrieving an audience measurement data packet from a television set in onier to identify the television 
program, (ii) detecting an audio code embedded in the television program in order to identify the television program, (iii) extracting an 
audio signature from the television program in order to identify the television program, (iv) identifying the television program through use 
of a software agent, and (v) selecting at least one of the retrieving means, the detecting means, the extracting means, and the software agent 
in order to identify the television program. 
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AUDIENCE MEASUREMENT SYSTEM FOR DIGITAL TELEVISION 

Technical Field of the Invention 

The present invention relates to the measure- 
ment of audiences of digitally broadcast television pro- 
5 gramraing and to the measurement of usage of other infor- 

mation services provided to the audiences. 

Backgrpuna q£ the Invention 

of the many approaches to measuring the usage 
of electronic entertainment equipment (commonly called 

10 '^tuning'' data) , one approach involves the addition of an 

identifying code to a radio or television program, the 
distribution of the identifying code with the program 
throughout the relevant broadcasting system, and the 
detection and interpretation of the identifying code 

15 when the broadcast signal is viewed or heard in a statis- 

tically selected monitoring site. An example of a system 
which implements this type of measurement approach may be 
found in the following patents: U.S. Patent No. 5,481,2- 
94 to Thomas et al., who describe, inter alia, identify- 

20 ing codes added to the vertical blanking interval of an 
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NTSC television broadcast; U.S. Patent No. 5,629,739 to 
Dougherty, who is particularly concerned with the addi- 
tion of an identifying code to a low energy portion of 
the audio spectrum of an NTSC signal; and U.S. Patent 
No. 5,404,377 to Moses, who teaches an audio encoding 
arrangement using signal masking to decrease the percep- 
tibility of the identifying code. The disclosures of 
U.S. Patent No. 5,481,294, U.S. Patent No. 5,629,739, and 
U.S. Patent No. 5,404,377 are herein incorporated by 
reference . 

•Another approach to measuring usage of elec- 
tronic entertainment equipment involves the extraction of 
a characteristic feature signature (or characteristic 
feature signature set) from the programming selected for 
viewing, and the storing of a time-stamped signature (or 
signature set) in a memory for subsequent transmission to 
a central data collection office where the signature (or 
signature set) is compared with corresponding reference 
signatures collected by the central office from known 
broadcasting sources. This approach is taught by Lert 
and Lu in U.S. Patent No. 
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4,677,466. The teaching of U.S. Patent No. 4,677,466 is 
herein incorporated by reference. 

Yet another approach to the measurement of the 
usage of electronic entertainment equipment has been that 
of comparing the viewed signal (or some component or 
artifact thereof) with all the signals available to a 
sampled household at the time the measurement is made. A 
review of apparatus and methods useful for this measure- 
ment approach is found in the teachings of Thomas et al . 
in U.S. Patent No. 5,629,739 and of Lu et al . in U.S. 
Patent No I- "5,594,934. The teachings in U.S. Patent No. 
5,594,934 and in U.S. Patent No. 5,629,739 are herein 
incorporated by reference. 

Wheeler, et al., in U.S. Application Serial No. 
08/786,270, filed on January, 22, 1997, teach a video 
signal source detection arrangement which determines the 
source of a video and/or audio signal being displayed by 
a television receiver. The arrangement implements a 
variety of tuning measurement approaches. For example, 
it can be used to read an identifying ancillary code 
transmitted with television programming, to obtain char- 
acteristic feature signatures from television program- 
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ming, or to obtain a signal that is matched with a con- 
temporary reference signal obtained by a television tuner 
controlled by measurement ecjuipment. The disclosure of 
the aforementioned U.S. Application Serial No. 08/786,270 
is herein incorporated by reference. 

Chan, in U.S. Application Serial No. 08/654,30- 
9, filed on May 28, 1996, teaches a sensor arrangement 
for non- intrusively obtaining a representation of video 
and synchronization signals from a television receiver. 
The disclosure of the aforementioned U.S. Application 
Serial No/ 08/654,309 is herein incorporated by refer- 
ence . 

Changes in the methods of measuring the recep- 
tion of television or radio programming are required 
because of a planned change-over from analog to digital 
broadcasting. In the U.S., the change-over is scheduled 
to be phased in by the year 2006, as documented in the 
Federal Communication Commission's Mass Media Docket 87- 
268, with particular reference to the Fifth Report and 
Order, FCC 97-116, April 3, 1997 and to the Sixth Report 
and Order, FCC 97-115, April 3, 1997. Some of the 
changes, and their respective impacts on approaches used 



wo 99/59275 



PCTAJS98/14286 



for measuring analog broadcasts, include the following: 
(1) digital broadcast techniques do not use a vertical 
blanking interval and, therefore, the program tracking 
and identifying codes that are written in the vertical 
blanking interval of analog broadcast signals will not be 
transmitted; (2) multiple signal formats and associated 
multiple display formats having, among other features, 
differing height-to-width ratios and different resolu- 
tions are allowed, so that existing video signal correla- 
tion methods used with analog broadcasts may essentially 
be disabled -because these methods depend on having the 
same pictorial feature appear at corresponding places on 
the measured and reference displays; (3) a broadcaster 
can transmit as many as six programs (arrayed as a se- 
quence of data packets, where each data packet is labeled 
as to which of the programs' data is carried therein) 
within an assigned 6 MHZ frequency band by trading off 
pictorial resolution for an increase in the number of 
programs so that (i) digital signal compression methods 
used to decrease the spectrum space required by a program 
destroy program identifying codes embedded in an origi- 
nal, high- resolution, program master, and (ii) determin- 
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ing which chemnel has been timed by a receiver does not 
uniquely identify a program being viewed if more than one 
program is being transmitted in that channel; and, (4) 
data other than television broadcasting may be co-trans- 
mitted in the same channel and, in some cases, it is 
expected that the other data will be related in some 
manner to the co-transmitted programming so that a viewer 
can interact with the TV programming (e.g., to obtain a 
program guide or detailed information on an advertised 
product, to automatically switch to a desired program, or 
to take part in an audience -participation program) . 

It may be noted that, although projected digi- 
tal signal compression and transmission methods destroy 
the types of video codes used in, and suggested for, the 
prior-generation analog broadcasting approaches, it is 
expected that less change will be encountered with re- 
spect to audio codes. That is, because the audio compo- 
nent of a television broadcast comprises much less infor- 
mation than does the video, there is much less to be 
gained by applying signal compression methods to audio. 
Nonetheless, it is expected that various audio compres- 
sion methods may be employed. These methods include 
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those defined by the Dolby AC- 3 or the European MUSICAN 
standards, or those defined by some yet -to-be-defined 
standard operable within the relatively open and flexible 
MPEG arrangement. Although the coding approaches taught 
in the aforementioned U.S. Patent No. 5,404,377 are known 
to survive the Dolby AC-3 and MUSICAN compression meth- 
ods, compression methods other than Dolby AC-3 and MUSIC- 
AN may destroy these codes. Moreover, although the chan- 
ge in technology may obviate the use of some of the code- 
s' such as those described above, other codes (e.g., 
digital data packet codes telling a receiving site which 
of several interleaved programs is associated with a 
given data packet) are expected to be broadcast both with 
television programming and with any co-transmitted data 
that are related to, and intended to be used in conjunc- 
tion with, one or more of the programs. 

In addition to changes in transmission equip- 
ment and signals, considerable changes are expected in 
reception equipment. Notable among these are both the 
use of set top signal converters (generally referred to 
as set top boxes) allowing a digital broadcast to be 
viewed on an analog receiver, and a projected convergence 
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of television receivers and computers • In the short 
term, for example, it is expected that digital receivers, 
configured as plug- in boards for personal computers, will 
be used to enable the display of digital television sig- 
5 nals on the computers. 

Another traditional measurement reflected in 
television audience reporting has been the determination 
of who actually viewed the program that was received by 
the equipment being monitored in a statistically selected 

10 monitoring site, A related measurement of interest is 

the identification of users of on-line services, such as 
on-line services provided over the Internet. Also, it is 
clear that measuring the members of an audience in a 
statistically selected monitoring site who are interac- 

15 tively using a television and/or accessing the Internet, 

and the information delivery apparatus used to deliver 
information to the members, will continue to be impor- 
tant . 

By far the most commonly used approach in mak- 
20 ing these measurements is that of interrogation, wherein 

the viewer and/or Internet user is asked to identify 
himself or herself as a member of the television audience 
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or as an Internet user. In connection with television 
viewing, this inquiry is usually done by means of an 
electronic prompting and data input device (commonly 
referred to as a Peoplemeter) associated with a monitored 
receiver in a statistically selected monitoring site. 
The member identification may also include age, sex, and 
other demographic data. It is common to store both the 
demographic data and the tuning data associated with each 
monitored receiver in the statistically selected monitor- 
ing site in store -and- forward equipment located within 
the monitoring site and to subsequently forward these 
data to a central office computer via a direct call over 
the public switched telephone network , or via the 
Internet, on a daily basis. 

Of particular interest in this area is the 
teaching of an audience interrogation arrangement dis- 
closed by McKenna et al., in U.S. Patent No. 4,816,904. 
According to this arrangement, a prompting message is 
displayed on a television screen overlaid on viewer se- 
lected programming by mixing the prompting message with 
the video signal being sent to the display. However, 
McKenna et al. do not teach a **tiling" arrangement where 
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the prompting message is displayed on a portion of a 
display separated from the portion of the display devoted 
to television picture. 

Many arrangements have been proposed in the 
computer arts for reliably identifying a user so that 
only selected individuals can secure access to some of 
the data on a computer or computer system. The most 
common of such arrangements is that of requiring the user 
to enter both his or her claimed identity along with a 
password. Other arrangements have included the use of 
various bioraetric techniques, such as image or voice 
recognition devices. Additionally, it is well known in 
the audience measurement arts to use computer-based image 
recognition in order to identify members of a viewing 
audience. Notable among teachings in this area is that 
by Lu in U.S. Patent No. 4,858,000. The teaching of this 
patent is herein incorporated by reference. 

Summary of the Invention 

According to one aspect of the present inven- 
tion, a television audience measurement system identifies 
a viewer selected television program from among a plural - 
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ity of television programs simultaneously broadcast in a 
broadcast channel. The television audience measurement 
system comprises a channel detector and a comparitor. 
The channel detector detects the broadcast channel in 
which the plurality of television programs is broadcast. 
The comparitor sequentially compares an audio component 
of the plurality of television programs broadcast in the 
detected broadcast channel with an audio signal of the 
viewer selected television program so as to identify the 
viewer selected television program. 

According to another aspect of the present 
invention, a television audience measurement system is 
provided for digital television equipment that is dis- 
posed in a statistically selected location. The televi- 
sion audience measurement system comprises a software 
agent and an interface and communication apparatus. The 
software agent is adapted to read, from a data packet 
contained in digital television programming, a datum 
identifying a television program, and the software agent 
is stored in memory associated with the digital televi- 
sion equipment. The interface and communication appara- 
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tus is adapted to transmit the identification datum to a 
remotely located central office. 

According to still another aspect of the pres- 
ent invention, a television audience measurement system 
identifies a viewer selected television program from 
among a plurality of television programs broadcast as a 
time division multiplexed sequence of data packets in a 
broadcast channel. The viewer selected television pro- 
gram is displayed on a television display in a statisti- 
cally selected location. The television audience mea- 
surement system comprises receiving means, acquiring 
means, recovering means, and comparing means. The re- 
ceiving means receives the time division multiplexed 
sequence of data packets in the broadcast channel. The 
acquiring means acquires an audio portion of the viewer 
selected television program. The recovering means recov- 
ers audio components respectively corresponding to the 
television programs contained in the sequence of data 
packets. The comparing means compares the audio compo- 
nents to the audio portion in order to determine the 
viewer selected television program. 
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According to yet another aspect of the present 
invention, a television audience measurement system mea- 
sures viewing of a television program viewed on digital 
television located in a statistically selected site and 
comprises detecting means, extracting means, a software 
agent, and selecting means. The detecting means detects 
an audio code embedded in the television program in order 
to identify the television program. The extracting means 
extracts an audio signature from the television program 
in order to identify the television program. The soft- 
ware agent 'is arranged to identify the television pro- 
gram. The selecting means selects at least one of the 
detecting means, the extracting mesms, and the software 
agent in order to identify the television program. 

According to a further aspect of the present 
invention, a method is provided to identify a television 
program selected by a viewer from a set of television 
programs broadcast as multiplexed data packets in a view- 
er selected broadcast channel. The viewer selected tele- 
vision program is displayed on a display portion of an 
apparatus tuned to the viewer selected broadcast channel. 
The method comprises the following steps: a) determining 

- 13- 
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the viewer selected broadcast channel; b) acquiring an 
audio portion of the viewer selected television program; 
c) selecting an audio component associated with one of 
the set of television programs broadcast in the viewer 
selected broadcast channel; d) comparing the audio por- 
tion with the audio component in order to determine 
whether the audio portion and the audio component match; 
e) if the audio portion and the audio component match, 
storing a tuning record from at least one of the audio 
portion and the audio component; and, f) if the audio 
portion and -the audio component do not match, selecting a 
different one of the television programs from the set of 
television programs and repeating steps (c) through (f ) 
until either a match is found or the set of television 
programs is exhausted. 

According to a still further aspect of the 
present invention, a method is provided to identify a 
viewer selected television program from among a plurality 
of time overlapped television programs broadcast in a 
viewer selected broadcast channel. The viewer selected 
television program is displayed in a first window of a 
multi -window television display- A data file, logically 
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related to the viewer selected television program, is 
also broadcast in the viewer selected channel so as to be 
time overlapped with the viewer selected television pro- 
gram. The viewer selected television program and the 
data file contain respective labels indicative of the 
logical relation- Data from the data file is displayed 
in a second window of the mult i -window display. The 
method comprises the following steps: a) reading, from 
the data file, the respective data file label and an 
identifying datum; b) finding, from the logical rela- 
tion, the* television program label associated with the 
respective data file label; and, c) storing a time- 
stamped record comprising the identifying datum. 

According to yet a further aspect of the pres- 
ent invention, a software agent is stored in memory asso- 
ciated with digital television equipment. The software 
agent is arranged to acquire television audience measure- 
ment data relative to the digital television equipment. 
The software agent comprises first, second, and third 
logging means. The first logging means logs a television 
program identification datum identifying a television 
program selected for viewing on the digital television 

-15- 
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equipment. The second logging means logs a co-transmit- 
ted identification datum associated with data co- trans- 
mitted with the television program selected for viewing 
on the digital television equipment. The third logging 
5 mecuis logs an Internet identification datum associated 

with an Internet task of the digital television equip- 
ment . 



Brief Description of the Drawing 

10 These and other features and advantages of the 

present invention will become more apparent from a de- 
tailed consideration of the invention when taken in con- 
junction with the drawings in which: 

Figure 1 is a schematic block diagram depicting 

15 typical digital television broadcasting and reception 

equipment ; 

Figure 2 is a schematic block diagram depicting 
an audio-based tuning measurement system which is usable 
with either digital or analog broadcasting and which is 
20 in accordance with a first embodiment of the present 

invention; 
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Figure 3 is a schematic block diagram depicting 
a tuning measurement system which employs a software 
agent resident in some of the monitored receiving equip- 
ment and which is in accordance with a second embodiment 
of the present inventions- 
Figure 4 is a schematic block diagram depicting 
the use of digital source detection in accordance with a 
third embodiment of the present inventions- 
Figure 5 is a schematic block diagram depicting 
the use of digital source detection in accordance with a 
fourth embodiment of the present inventions- 
Figure 6 is a schematic block diagram depicting 
the use of data ports on various television viewing 
equipment in order to measure television related data in 
accordance with a fifth embodiment of the present inven- 
tion; and. 

Figure 7 is a flow chart depicting the opera- 
tion of a software agent of the present invention. 

Detailed Description of the Invention 

A typical digital television broadcasting envi- 
ronment, which is planned during a transition period in 
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which both digital and analog programs are transmitted, 
may comprise some combination of the elements depicted in 
Figure 1. Network programming, and other non-local 
programming, is provided to a local television station 10 
from a broadcast station over a communication medium 12, 
such as a satellite, or terrestrial digital transmission 
lines, or the like. Local programming is also provided 
to the local television station 10 from some mixture of 
high definition television (HDTV) cameras 14 (which may 
employ a variety of formats) , standard definition digital 
video cameras 16 (which may employ a variey of resolu- 
tions and aspect ratios) , and analog sources comprising 
both analog video cameras 18 (e.g., those compatible with 
the prior generation NTSC broadcast standard) , and cinema 
film sources 20 that have historically been converted to 
the NTSC standard by means of well known telecine 22 . 

The signals from the analog video cameras 18 
and from the cinema film sources 20 are input to an A/D 
converter 24 having an appropriate digital output. The 
outputs from the A/D converter 24, from the communication 
medium 12, from the high definition television (HDTV) 
cameras 14, and/or from the standard definition digital 

-18- 
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video cameras 16 are provided to a switching and routing 
apparatus 26 . The switching and routing apparatus 2c 
passes selected ones of the signals provided zo it throu- 
gh digital compression equipment 28 prior to being broad- 
cast from the station's transmitter 30. The station's 
transmitter 30 may also include an antenna, a satellite 
dish, a cable modem, or the like. 

In addition, a cable headend 31 (which is shown 
as a single cable provider but which may represent a 
plurality of cable providers) receives signals over the 
air, from 'satellite distribution, or the like, and dis- 
tributes the signals (with or without reformatting) by 
way of regular cables, optical fibers, microwaves, or the 
like. 

The digital transmission from the local televi- 
sion station 10 and/or from the cable headend 31 may be 
directly received and displayed in a statistically se- 
lected monitoring site 32 by a digital television set 34. 
The digital transmission also may be directly received 
and converted by a digital converter 36 (commonly config- 
ured as a set top box or STB) into an analog output (e.g- 
in the NTSC format) for use by an analog television 
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set 38 purchased by a viewer prior to the change-over no 
digital programming. The digital transmission further 
may be directly received and displayed by a personal 
computer 4 0 that has a digital receiver 4 2 providing a 
baseband video output signal that the personal computer 
40 can show in one of the windows 44 defined on its dis- 
play unit 46. The digital receiver 42, for example, 
could be a plug in board of the personal computer 4 0 or 
integrated on the motherboard. The personal computer 
also may have speakers 47, a keyboard 48, and a mouse 50 
or other pointing device known in the art. 

Moreover, the digital transmission may be di- 
rectly received and converted by a digital converter 52 
to a form useable by a digital television set 54. For 
example, the digital converter 52 may be used for pay- 
per-view television programming, video-on-demand televi- 
sion programming, satellite programs, and/or other ser- 
vices provided to the statistically selected monitoring 
site 32. The digital transmission also may be directly 
received and converted by a digital converter 56 to a 
form useable by a simple monitor 58. In either of these 
two cases, the digital television viewing can be measured 

-20- 
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by means of metering technologies similar to the set top 
box and an analog television configuration described 
below. 

In addition to the direct reception of the 
digital transmission by apparatus dedicated solely to 
that purpose (e.g., the digital television set 34 and/or 
the personal computer 40) , it is expected that a variety 
of yet-to-be-defined consumer electronic apparatuses will 
be configured that incorporate at least some of the fea- 
tures of both a digital or analog television receiver and 
a home computer. 

A first embodiment 6 0 of the present invention, 
which measures tuning to digital television programming 
received in a statistically selected monitoring site 62, 
is depicted schematically in Figure 2. The first embodi- 
ment 60 measures such tuning by detecting program identi- 
fication codes and/or extracting signatures from the 
audio portion of the television signal to which a re- 
ceiver is tuned. This audio portion is detected by a 
detector 64 in the case of a digital television set 66, 
and by a detector 68 in the case of an analog television 
set 70 . The detector 64 may detect the audio portion of 
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a program to which the digital television ser €6 is tuned 
by non- intrusively detecting the sound provided by a 
speaker 72 of the digital television set 66 {in which 
case the detector 64 may be a microphone) or by intru- 
sively detecting the audio signal from a direct connec- 
tion to the speaker, the audio output terminals, or the 
audio processing circuitry of the digital television set 
66. Similarly, the detector 68 may detect the audio 
portion of a program to which the analog television set 
70 is tuned by non- intrusively detecting the sound pro- 
vided by a speaker 74 of the digital television set 66 or 
by intrusively detecting the audio signal from a direct 
connection to the speaker, the audio output terminals, or 
the audio processing circuitry of the analog television 
set 70, The analog television receiver 70 is provided 
with a digital converter 76, which may be similar to the 
digital converter 36 of Figure 1. 

The audio portion of a television program may 
also be detected by a detector 78 when the television 
program is played by a personal computer 80 on a monitor 
82 and speakers 84. The detector 78 may detect the audio 
portion of a program to which the personal computer 80 is 
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tuned by non- intrusively detecting the sound provided by 
the speakers 84 or by intrusively detecting the audio 
signal from a direct connection to the speaker, the audio 
output terminals, or the audio processing circuitry of 
the personal computer 80. 

As has been taught by Thomas et al., in U.S. 
Patent No. 5,629,739, a representation of the audio por^ 
tion picked up by the detector 64 can be analyzed by a 
site unit (SU) 86 and/or a home unit (HU) 88 (which may 
be provided with data storage and forwarding unit capa- 
bility) within the statistically selected monitoring site 
62 in order to determine if the audio portion from the 
digital television set 66 contains television program 
identification codes. Alternatively, the representation 
of the audio portion picked up by the detector 64 can be 
analyzed at a remotely located data collection central 
office 90 in order to determine if the audio portion from 
the digital television set 66 contains such television 
program identification codes. The remotely located data 
collection central office 90 is schematically depicted as 
a computer based data collection central office that 
interchanges data with the home unit 88 over a communica- 

-23- 



W0 99/5W75 



PCT/US98/14286 



tion network 92, such as a public switched telephone 
network, the Internet, or the like* 

Similarly, a representation of the audio por- 
tion picked up by the detector 68 can be analyzed by a 
site unit 94 and/or the home unit 88 in order to deter- 
mine if the audio portion from the analog television set 
70 contains television program- identifying codes, and a 
representation of the audio portion picked up by the 
detector 78 can be analyzed by a site unit 96 and/or the 
home unit 88 in order to determine if the audio portion 
from the personal computer 80 contains television 
program- identifying codes. Alternatively, the represen- 
tation of the audio portions picked up by the detectors 
68 and 78 can be analyzed at the remotely located data 
collection central office 90 in order to determine if the 
audio portions from the analog television set 70 and the 
personal computer 80 contain television program identifi- 
cation codes. 

Moreover, a person identifier 98 may be pro- 
vided in order to identify the persons watching televi- 
sion programming on the digital television set 66. The 
person identifier 98 may be video camera, an IR camera, 
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or the like. When such equipment is available in the 
statistically selected monitoring site 62, the site unit 
86 may employ known head location and face recognition 
software (e.g., as taught by Lu in U.S. Patent No. 
4,858,000) for the identification of the viewing persons 
and for the collection of other demographic data. Simi- 
larly, person identifiers may be provided in order to 
identify the persons watching television programming on 
the analog television set 70 and on the personal computer 
80 . 

Instead of analyzing the representations of the 
audio portions picked up by the detectors 64, 68, and 78 
in order to determine if the audio portions from the 
corresponding digital television set 66, analog televi- 
sion set 70, and personal computer 80 contain television 
program- identifying codes, characteristic signatures can 
be extracted from these representations for subsequent 
comparison, either in the statistically selected monitor- 
ing site 62 or in the central office 90, with previously 
collected reference signatures in order to identify the 
television programs to which the digital television set 
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66, the analog television set 70, and the personal com- 
puter 80 are tuned. 

Programs played by VCRs on the digital televi- 
sion set 66, on the analog television set 70, and on the 
personal computer 80 can be identified using either the 
code or the signature technique. Thus, if the programs 
being played carry program identification codes that are 
preserved through the process of recording and replay in 
the case of time shifted viewing, or that are recorded in 
the programs stored on tapes to be rented or sold, the 
program identification codes can be detected at the time 
that the programs are played so that the audiences of the 
programs may be measured. However, if the programs being 
played are non-encoded programs, or if the program iden- 
tification codes are lost (e.g., during the recording 
process) , then the embodiments of the present invention 
described' herein can determine the programs being viewed, 
when the tapes are played, by extracting characteristic 
signatures from these programs and by matching the ex- 
tracted signatures to reference signatures stored in a 
reference signature database. In this case, searching 
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can be optimized by the use of well knovm hashing techniques 

Although the present invention may use either 
the code technique or the signature technique in order to 
measure audiences of programs played on digital televi- 

5 sion equipment, the present invention may use both of 

these two techniques so that one of the techniques backs 
up the other in the event that the one technique is un- 
able to identify a viewer selected television program. 
For example, signatures are extracted and matched in 

10 order to identify a viewer selected television program 

only if identifying codes cannot be detected. As a fur- 
ther alternative, the present invention may use both of 
these two techniques simultaneously. Demographic data 
may also be collected in connection with the present 

15 invention. 

It may be noted that, although the first em- 
bodiment 60 measures the great majority of the overall 
audience, it is not arranged to measure interactive fea- 
tures (e.g., CO- transmitted advertising detail) that are 
20 expected to be part of the digital broadcasting environ- 

ment. Therefore, an important feature of the following 
embodiment is that it measures interactive features. 
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Accordingly, a second embodiment 100 of the 
present invention is provided as shown in Figure 3. The 
second embodiment 100 measures tuning to digital televi- 
sion programming in connection with a statistically se- 
lected monitoring site 102. As will be apparent from the 
discussion below, this second embodiment 100 may include 
some or all of the features of the first embodiment 60, 
as well as the new features described below in connection 
with the second embodiment 100. 

According to this second embodiment 100, tuning 
data, and' possibly demographic data, are collected from 
digital television reception equipment that includes 
processing equipment preferably having adequate non-vola- 
tile or battery-backed memory available so that an algo- 
rithm, hereinafter referred to as a software agent, can 
be stored thereon. The digital television reception 
equipment of the second embodiment 100 may include (i) a 
personal computer 104 which may be similar to the per- 
sonal computer 40 of Figure 1, (ii) a digital converter 
106 which may be similar to the digital converter 36 of 
Figure 1, (iii) an analog television set 108 operating in 
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response to the digital converter 106. and (v) a digital 

television set 110. 

one or more software agents of the second em- 
bodiment 100. as disclosed in greater detail hereinafter, 
monitor operating tasks in order to create time-stamped 
records containing tuning and/or other data regarding 
operation of the particular pieces of equipment in which 
the software agents are resident. Such monitored operat- 
ing tasks may be. for example, messages passed from a 
remote or other control operated by a user to a tuner or 
other program selector in order to select a television 
program being broadcast in a channel, messages passed 
from a controller of processing equipment to a tuner in 
order to select a television program being broadcast in a 
channel, or the like. Although software agents are ex- 
pected to be able to effectively eavesdrop on data traf- 
fic passing though their hosts' data ports, the use of 
software agents would not fully measure tuning at viewing 
sites unless each item of television equipment at those 
sites has a respective software agent within it so that a 
complete set of operating tasks can be captured. 
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Accordingly, a software agent 112 is included 
in the digital converter 106. The software agent 112 
monitors operating tasks in order to create time-stamped 
records containing tuning and/or other data regarding 
operation of the digital converter 106 in which the soft- 
ware agent 112 is resident. Thus, the software agent 112 
detects, for example, a message relating to selection of 
a television program being broadcast in a selected chan- 
nel. From this detected message, the identity of the 
selected television program may be determined. The soft- 
ware agent -112 is also arranged to eavesdrop on data 
traffic passing though its host's input/output data port 
114, and to communicate over the input /output data port 
114. Accordingly, programs to which the analog televi- 
sion set 108 is tuned may be detected through use of the 
software agent 112. The software agent 112 may be down- 
loaded over a communication medium 116 to the digital 
converter 106 by use of the input/output data port 114. 
Alternatively, the software agent 112 may be a plug in, 
may be resident on a floppy disk, or the like. 

Similarly, a software agent 118 is included in 
the digital television set 110. The software agent 118 
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monitors operating tasks associated with the digital 
television set 110 in order to create time-stamped re- 
cords containing tuning and/or other data regarding oper- 
ation of the digital television set 110 in which the 
software agent 118 is resident. The software, agent 118 
is also arranged to cooperate with its host's in- 
put/output data port 120. A software agent 122 is in- 
cluded in the personal computer 104. The software agent 
122 monitors operating tasks associated with the personal 
computer 104 in order to create time-stamped records 
containing tuning and/or other data regarding operation 
of the personal computer 104 in which the software agent 
122 is resident. The software agent 122 is also arranged 
to cooperate with its host's input/output data port 124. 

Moreover, a person identifier 126 may be pro- 
vided in order to identify the persons watching televi- 
sion programming on the personal computer 104 . As in the 
case of the person identifier 98. the person identifier 
126 may be video camera, an IR camera, or the like, and 
may be provided with any or all of the viewing equipment 
shown in Figure 4. 
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The software agents 112, 118, and 122 may be 
telecommunication enabled so that television audience 
data from all the viewing sites in the statistically 
selected monitoring site 102 can be communicated, via the 
Internet, a public telephone system, or the like, to a 
locally located or remotely located intermediate data 
collector 128 and then to a remotely located central 
office 130 through a communication channel 170. The 
intermediate data collector 128 can be flexibly located 
either at the same statistically selected monitoring site 
102, or at a remotely located site 166, or at the central 
office 130. For this purpose, the input/output data 
ports 114, 120, and 124 may be serial ports, parallel 
ports, universal serial buses (USB), firewires (according 
to IEEE 1394), or the like. The data transfer may, for 
example, be structured as a background task and take the 
form of a sequence of data packets, each of which is e- 
mailed or otherwise communicated to the remotely located 
intermediate data collector 128 during pauses in a user's 
on line sessions or at any other time. The intermediate 
data collector 128. for example, may be a computerized 
data collector or an Internet service provider. Altema- 
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tively, the intermediate data collector 128 may be a home 
unit, in which case, the intermediate data collector 128 
may be located within the statistically selected monitor- 
ing site 102 and may itself be arranged to communicate 
5 tuning and demographic data via e-mail or other communi- 

cation to the central office 130. 

The software agent 122 additionally may monitor 
the personal computer 104 for interactive uses such as 
Internet usage (e.g., through logging of Universal Re- 
10 source Locators, URLs) , application software usage, and 

television viewing supported by the personal computer 
104. Similarly, to the extent that the digital converter 
106 and the digital television set 110 are web-enabled, 
the software agents 114 and 118 may monitor interactive 
15 uses such as Internet usage (e.g., through logging of 

URLs) and application software usage, as well as televi- 
sion viewing. 

Because not all equipment that is encountered 
by a research firm doing television audience research, at 
20 least during the early part of the digital television 

broadcasting era, will be compatible with the software 
agent approach discussed above, the second embodiment 100 
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of the measurement system of the present invention may be 
arranged to include many or all of the features found in 
the first embodiment 60. 

As described above, the first embodiment 60 
measures the great majority of the overall audience by 
detecting the programs to which receivers are tuned by 
detecting program identification codes embedded in the 
programs, and/or by extracting signatures of the programs 
being viewed and by comparing the extracted signatures to 
reference signatures. Moreover, the second embodiment 
100 measures not only the overall audience, but also 
interactive features provided with the programming. The 
embodiments described below are specifically arranged to 
measure, in an efficient manner, the digital television 
viewing data where multiple television programs are pres- 
ent in a common digital transmission channel (e.g.. of 6 
MHZ) , particularly where a digital television set does 
not have an accessible operating system with which a 
software agent can reside. The first embodiment 60 is 
also useful where there is no I/O interface on the tele- 
vision set so that there is no easy way to read data from 
the television set. 
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Figure 4 depicts a third embodiment 200 of the 
invention in which a statistically selected monitoring 
site 202, during the transition period when both analog 
and digital broadcasting occur, includes, for example, a 
first viewing site 204, a second viewing site 206, and a 
third viewing site 208. The first viewing site 204 in- 
cludes a digital converter 210 connected to an analog 
television set 212. The audio portion of a program to 
which the analog television set 212 is tuned is detected 
by a detector 214. The detector 214 may detect this 
audio portion by non- intrusively detecting the sound 
provided by a speaker 216 of the analog television set 
212 or by intrusively detecting the audio signal from a 
direct connection to the analog television set 212. 

A probe 218 is provided to detect the interme- 
diate frequency signal of the digital converter 210. A 
signal processor 220 demodulates the detected intermedi- 
ate frequency signal to the baseband bit stream contained 
in the detected intermediate frequency signal, and then 
demultiplexes the baseband bit stream to recover the 
audio portion of the baseband bit stream. 
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A comparitor 222 may be arranged to first de- 
termine whether there is a program identification code in 
the audio detected by the detector 214. If there is a 
program identification code in the audio detected by the 
detector 214, this program identification code identifies 
the program to which the analog television set 212 is 
tuned and is, therefore, logged. If there is no program 
identification code in the audio detected by the detector 
214, the comparitor 222 then scans the audio portion of 
the baseband bit stream provided by the signal processor 
220 in order to detect the audio in a first program con- 
tained in the intermediate frequency signal detected by 
the probe 218. The comparitor 222 compares this audio to 
the audio data from the detector 214. If there is a 
match, the program to which the digital converter 210 is 
tuned is known from the information provided by the sig- 
nal processor 220. If there is no match, the comparitor 
222 then scans the audio portion of the baseband bit 
stream provided by the signal processor 220 in order to 
detect the audio in a second program contained in the 
intermediate frequency signal detected by the probe 218, 
and so on until the comparitor 222 detects a match. Once 
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a match is found, the major and minor channel numbers of 
the matched channel are known, and the corresponding 
program identifying ancillary code can be read from the 
corresponding data packet. 

The second viewing site 206 includes a digital 
television set 224 . The audio portion of a program to 
which the digital television set 224 is tuned is detected 
by a detector 226. The detector 226 detects this audio 
portion by non- intrusively detecting the sound provided 
by a speaker 228 of the digital television set 224 or by 
intrusively -detecting the audio signal from a direct 
connection to the digital television set 224* 

A probe 230 is provided to detect the interme- 
diate frequency signal of the digital television set 224. 
A signal processor 232 operates similarly to the signal 
processor 220 in order to provide the audio portion of 
the baseband bit stream in the intermediate frequency 
signal detected by the probe 230. A comparitor 234 oper- 
ates similarly to the comparitor 222 in order to first 
determine whether there is a program identification code 
in the audio detected by the detector 226 and, if there 
is not, to scan the audio portion of the baseband bit 
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stream provided by the signal processor 232 until a match 
is found. 

The third viewing site 208 includes a personal 
computer 236. The audio portion of a program to which 
the personal computer 236 is tuned is detected by a de- 
tector 238. The detector 238 detects this audio portion 
by non- intrusively detecting the sound provided by speak- 
ers 240 of the personal computer 236 or by intrusively 
detecting the audio signal from a direct connection to 
the personal computer 236. 

A. probe 242 is provided to detect the interme- 
diate frequency signal of the personal computer 236. A 
signal processor 244 operates similarly to the signal 
processor 220 in order to provide the audio portion of 
the baseband bit stream in the intermediate frequency 
signal detected by the probe 242. A comparitor 246 oper- 
ates similarly to the comparitor 222 in order to first 
determine whether there is a program identification code 
in the audio detected by the detector 238 and, if there 
is not, to scan the audio portion of the baseband bit 
stream provided by the signal processor 244 until a match 
is found. 
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The cotnparitors 222, 234, and 246 provide their 
program identification codes from the audio or from the 
data packet, and/or match information to a home unit 248, 
which may be provided with data storage and forwarding 
unit capability, within the statistically selected moni- 
toring site 202. A remotely located data collection 
central office 250 is schematically depicted as a com- 
puter based data collection central office that inter- 
changes data with the home unit 248 over a communication 
network 252, such as a public switched telephone network, 
the Internet-, or the like. Moreover, a person identifier 
254 may be provided in order to identify the persons 
watching television programming on the digital television 
set 224. As in the case of the person identifier 98, the 
person identifier 254 may be video camera, an IR camera, 
or the like, and may be provided for any or all of the 
viewing equipment shown in Figure 4 , 

Alternatively, instead of using intermediate 
frequency probes for the probes 218, 230, and 242, local 
oscillator probes may be used. A local oscillator probe 
may be employed to initially identify the channel tuned 
by its associated television receiver, which offers a 
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significant speed advantage in the measurement. That is, 
because channel detection apparatus normally operates 
much faster than does audio signal comparison apparatus, 
the program determination process can be carried out in a 
hierarchical fashion in which the broadcast channel is 
first determined from the local oscillator signal by the 
signal processors 220, 232, and 244 (assuming that a 
program identification code is not first detected) . 
Then, the comparitors 222, 234, and 246 sequentially tune 
to the programs co- transmitted in the detected broadcast 
channel and compare the audio portion of the tuned pro- 
gram to the audio data from detectors 214, 226, and 238 
in order to identify the program in the broadcast chan- 
nel. In this way, the comparitors 222, 234, and 246 are 
not required to sequentially tune through all of the 
programs in all of the available channels in order to 
determine the tuned program. 

A still further alternative is provided by a 
fourth embodiment 300 of the invention shown in Figure 5. 
The fourth embodiment 300 is used in a statistically 
selected monitoring site 302 having a first viewing site 
304, a second viewing site 306. and a third viewing site 
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308. The first viewing site 3 04 includes a digital con- 
verter 310 connected to an analog television set 312. 
The audio portion of a program to which the analog tele- 
vision set 312 is tuned is detected by a detector 314. 
5 The detector 314 may detect this audio portion by non- 

intrusively detecting the sound provided by a speaker 316 
of the analog television set 312 or by intrusively de- 
tecting the audio signal from a direct connection to the 
audio processing circuitry of the analog television set 
10 312. 

A scanning receiver 318 is arranged to sequen- 
tially tune to each of the programs carried in each 
broadcast channel to which the digital converter 310 may 
be tuned, and to provide an audio portion of each such 

15 program to a comparitor 320. The comparitor 320 may be 

arranged to first determine whether there is a program 
identification code in the audio detected by the detector 
314. If there is a program identification code in the 
audio detected by the detector 314. this program identi- 

20 fication code is logged because it identifies the program 

to which the analog television set 312 is tuned. If 
there is no program identification code in the audio 
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detected by the detector 314, the comparitor 320 then 
compares the audio portion of a first program provided by 
the scanning receiver 318 to the audio data from the 
detector 314. If there is a match, the program to which 
5 the digital converter 310 is tuned is known from the 

information provided by the scanning receiver 318. Once 
a match is found, the major and minor channel numbers of 
the matched channel are known, and the corresponding 
program identifying ancillary code can be read from the 
10 corresponding data packet. 

If there is no match, the comparitor 320 com- 
pares the audio portion of a second program provided by 
the scanning receiver 318 to the audio data from the 
detector 314. and so on until the comparitor 320 detects 
15 a match. A demographic input 322 may be used to provide 

demographic data to the comparitor 320 which couples this 
demographic data, and time, with the program to which the 
digital converter 310 is tuned as a tuning record. The 
demographic input 322, for example, may be the well know 
20 Peoplemeter with which people currently identify them- 

selves manually when they are viewing a program displayed 
on a television set. 
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The second viewing site 306 includes a digital 
television set 324 . The audio portion of a program to 
which the digital television set 324 is tuned is detected 
by a detector 326. The detector 326 detects this audio 
portion by non- intrusively detecting the sound provided 
by a speaker 328 of the digital television set 324 or by 
intrusively detecting the audio signal from a direct 
connection to the digital television set 324. 

A scanning receiver 330 is arranged to sequen- 
tially tune to each of the programs carried in each 
broadcast channel to which the digital television set 324 
may be tuned, and to provide an audio portion of each 
such program to a comparitor 332. The comparitor 332 may 
be arranged to first determine whether there is a program 
identification code in the audio detected by the detector 
326. If there is a program identification code in the 
audio detected by the detector 326, this program identi- 
fication code is logged because it identifies the program 
to which the digital television set 324 is tuned. If 
there is no program identification code in the audio 
detected by the detector 326. the comparitor 332 then 
compares the audio portion of a first program provided by 
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the scanning receiver 330 to the audio data from the 
detector 326. If there is a match, the program to which 
the digital television set 324 is tuned is known from the 
information provided by the scanning receiver 330. If 
there is no match, the comparitor 332 compares the audio 
portion of a second program provided by the scanning 
receiver 330 to the audio data from the detector 326, and 
so on until the comparitor 332 detects a match. 

Moreover, a person identifier 334 may be pro- 
vided in order to identify the persons watching televi- 
sion programming on the digital television set 224. As 
in the case of the person identifier 98, the person iden- 
tifier 254 may be video camera, an IR camera, or the 
like, and may be provided for any or all of the viewing 
equipment shown in Figure 4. The comparitor 332 couples 
the identities of the persons watching the digital tele- 
vision set 324, and time, with the program to which the 
digital television set 324 is tuned as a tuning record. 

The third viewing site 308 includes a personal 
computer 336. The audio portion of a program to which 
the personal computer 336 is tuned is detected by a de- 
tector 338- The detector 338 detects this audio portion 
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by non-intrusively detecting the sound provided by speak- 
ers 340 of the personal computer 336 or by intrusively 
detecting the audio signal from a direct connection to 
the personal computer 336. 

A scanning receiver 342 is arranged to sequen- 
tially tune to each of the programs carried in each 
broadcast channel to which the personal computer 336 may 
be tuned, and to provide an audio portion of each such 
program to a comparitor 344 . The comparitor 344 may be 
arranged to first determine whether there is a program 
identification code in the audio detected by the detector 
338. If there is a program identification code in the 
audio detected by the detector 338, this program identi- 
fication code is logged because it identifies the program 
to which the personal computer 336 is tuned. If there is 
no program identification code in the audio detected by 
the detector 338, the comparitor 344 then compares the 
audio portion of a first program provided by the scanning 
receiver 342 to the audio data from the detector 338. If 
there is a match, the program to which the personal com- 
puter 336 is tuned is known from the major and minor 
channel numbers indicated by the match. If there is no 
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match, the comparitor 344 compares the audio portion of a 
second program provided by the scanning receiver 342 to 
the audio data from the detector 338, and so on until the 
comparitor 344 detects a match. The comparitor 344 cou- 
ples time with the program to which the personal computer 
336 is tuned as a tuning record. 

The comparitors 222, 234, and 246 provide their 
tuning records to a home unit 346, which may be provided 
with data storage and forwarding unit capability, within 
the statistically selected monitoring site 302. A re- 
motely loca'ted data collection central office 348 is 
schematically depicted as a computer based data collec- 
tion central office that interchanges data with the home 
unit 346 over a communication network 350, such as a 
public switched telephone network, the Internet, or the 
like . 

It is likely that many, if not all, digital 
television sets, digital converters, and like digital 
equipment will have data ports that will provide output 
viewer- selected program information for use in consumer 
devices connected to the data ports. In connection with 
digital televisions, a data port is likely to provide, 
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for example, the full major channel ATSC bitstream and 
information on virtual channels that may include analog 
or digital major channel selection, program (minor chan- 
nel) selection, physical source (TV, VCR, DVD, etc.) 
identification, and the like. In connection with digital 
converters (i.e., set top boxes), a data port is likely 
to provide, for example, the full ATSC bitstream and 
information on virtual channels that may include analog 
or digital major channel selection, program (minor chan- 
nel) selection, vertical blanking interval data rein- 
serted on-.tlie correct line and field of analog signals, 

and the like. 

Accordingly, a fifth embodiment 400 of the 
present invention is provided as shovm in Figure 6. The 
fifth embodiment 400 measures tuning to digital televi- 
sion programming in connection with a statistically se- 
lected monitoring site 402. The fifth embodiment 400 may 
include some or all of the features of the previously 
described embodiments. 

The fifth embodiment 400 permits tuning data, 
and possibly demographic data, to be collected from digi- 
tal television reception equipment that includes (i) a 
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personal computer 404, (ii) a digital converter 406, 
(iii) an analog television set 408 operating in response 
to the digital converter 406, and (v) a digital televi- 
sion set 410. 

The digital converter 406 has a data port 412. 
A site unit 414 reads and stores the full ATSC bitstream, 
the analog or digital major channel selection, the pro- 
gram selection, vertical blanking interval data, and/or 
the like data available at the data port 412 in order to 
determine the program selected for viewing on the analog 
television set 408 as well as any other desired and 
available information. Moreover, the site unit 414 may 
be arranged to collect demographic data related to the 
viewers in the audience of the program selected for view- 
ing on the analog television set 408. For example, the 
site unit 414 may collect demographic data from a person 
identifier, a Peoplemeter, or the like, as discussed 
above. All such data may be uploaded over a communica- 
tion medium 416, which may be a wire, a power line, and 
IR link, an RF modem, or the like. 

Similarly, the digital television set 410 has a 
data port 418. A site unit 420 reads and stores the full 

-48- 



wo 99/59275 



PCTAJS98/14286 



major channel ATSC bitstream, the major channel selec- 
tion, the program selection, physical source identifica- 
tion, and the like data available at the data port 418 in 
order to determine the program selected for viewing on 
5 the digital television set 410 as well as any other de- 

sired and available information. Moreover, the site unit 
420 may be arranged to collect demographic data related 
to the viewers in the audience of the program selected 
for viewing on the digital television set 408. For exam- 
10 pie, the site unit 420 may collect demographic data from 

a person identifier, a Peoplemeter, or the like, as dis- 
cussed above. All such data may be uploaded over a com- 
munication medium 421. 

Also, the personal computer 404 has a data port 
15 422. A site unit 424 reads and stores the full major 

channel ATSC bitstream, the major channel selection, the 
program selection, and the like data available at the 
data port 422 in order to determine the program selected 
for viewing on the personal computer 404 as well as any 
20 other desired and available information. Moreover, a 

person identifier 426 may be provided in order to iden- 
tify the persons watching television programming on the 
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personal computer 404. As in the case of the person 
identifier 98, the person identifier 426 may be video 
camera, an IR camera, or the like, and may be provided 
with any or all of the viewing equipment shown in Figure 
6. All program and demographic data may be uploaded over 
a communication medium 427. 

The site units 414, 420, and 424 may be tele- 
communication enabled so that television audience data 
from all the viewing sites in the statistically selected 
monitoring site 402 can be communicated, via the Interne- 
t, a public telephone system, or the like, to a locally 
located or remotely located intermediate data collector 
428 and then to a remotely located central office 430 
through a communication channel 470. The intermediate 
data collector 428 can be flexibly located either at the 
same statistically selected monitoring site 402, or at a 
remotely located site 466, or at the central office 430. 
For this purpose, the site units 414, 420, and 424 may be 
provided with serial ports, parallel ports, universal 
serial buses (USB), firewires (according to IEEE 1394), 
or the like. The intermediate data collector 428, for 
example, may be a computerized data collector or an 
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Internet service provider. Alternatively, the intermedi- 
ate data collector 428 may be a home unit, in which case, 
the intermediate data collector 428 may be located within 
the statistically selected monitoring site 402 and may 
5 itself be arranged to communicate tuning and demographic 

data via e-mail or other communication to the central 
office 430. 

A software agent 500 is shown in Figure 7. The 
software agent 500 can be used for any of the software 

10 agents 112, 118, and 122. As shown at a block 502 of the 

software agent 500, the software agent 500 copies the 
current operating task. If the current operating task 
uses a packet of television programming as determined at 
a block 504, the software agent 500 at a block 506 deter- 

15 mines whether the data packet has a decodable packet 

label including a decodable program identification code, 
name, or other indicia. This program identification data 
packet is expected to be a feature of digital television 
programming, particularly if multiple programs are packed 

20 simultaneously as minor channels in a broadcast channel 

in a digital broadcasting environment, and is expected to 
identify the program tuned by the monitored television. 
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15 



20 



If the data packet does not have a decodable 
packet label, the software agent 500 at a block 508 caus- 
es the ON state of the monitored viewing equipment to be 
logged, provided that this ON state is a new status for 
the monitored viewing equipment and the monitored viewing 
equipment is ON. The balance of the measurement system 
(e.g., the audio signal methods employed by any of the 
other embodiments 60, 200, and 300 described above) is 
relied on in order to identify the tuning. Program flow 
then returns to the block 502 to await the next operating 
task . 

on the other hand, if the data packet does have 
a decodable packet label, the software agent 500 deter- 
mines at a block 510 whether the packet label from the 
data packet of the current operating task is the same as 
the packet label previously determined by the software 
agent 500. If so, there is no need to store the packet 
label unless it is merely indicative of continued viewing 
to an already identified television program. Accord- 
ingly, program flow returns to the block 502 to await the 
next operating task. However, if the packet label from 
the data packet of the current operating system task is 
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not the same as the packet label previously determined at 
the block 506, the software agent 500 at a block 512 logs 
the identification of the television program as contained 
in the program label and also logs the time. Program 
flow then returns to the block 502 to await the next 
operating task. 

If the current operating task does not use a 
packet of television programming as determined at a block 
504, the software agent 500 at a block 514 determines 
whether the current operating task is a command to other 
monitored' equipment (e.g., a command to the digital tele- 
vision receiver 110 to tune a different channel and to 
select a particular one of the N programs being broadcast 
in that channel) . If the current operating task is a 
command to other monitored equipment, the software agent 
500 at a block 516 logs as much detail as is available 
(e.g-, the command itself, the identity of the issuer of 
the command, and the identity of the recipient of the 
command) in memory, and program flow returns to the block 
502. 

If the current operating task is not a command 
to other monitored equipment, the software agent 500 at a 
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block 518 determines if co-transmitted data (i.e., data 
that is related to one of the television programs being 
broadcast in a channel and that is transmitted during the 
same time interval as the program) has been selected by a 
viewer. Co- transmitted data may comprise a guide to 
other available television programming, catalog- like 
details on products being advertised on the co-transmit- 
ted program, and the like. Some of these proposed uses 
of co-transmitted data will be configured so that one can 
infer what program is being viewed from a URL or other 
label that logically links the co-transmitted data to the 
television program. Accordingly, the software agent 500 
at a block 520 searches the header portion of a task for 
such URL or other label, and logs such URL or other label 
that is found. Thereafter, program flow returns to the 
block 502. 

If co-transmitted data has not been selected by 
a viewer, the software agent 500 at a block 522 deter- 
mines if the user of the equipment has directed his or 
her browser, or other communication software, to access 
an internet page (assuming that the monitored viewing 
equipment being monitored is configured so that it has 
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access to the Internet or other on-line service) . If the 
user of the equipment has directed his or her browser, or 
other communication software, to access an internet page, 
the software agent 500 at a block 524 searches for a URL 
or other label, and logs such URL or other label that is 
found. Thereafter, program flow returns to the block 
502. If the user of the equipment has not directed his 
or her browser, or other communication software, to ac- 
cess an internet page, program flow returns to the block 
502. 

As an example of the use of a software agent as 
describe above, consider the case of a person in a sta- 
tistically selected monitoring site who is viewing a 
digital television program and who sees an interesting 
product being advertised. It is expected that the person 
will have the capability to request more data on that 
product by means of a simple mouse-click operation. This 
data, for example, will either be a subset of the data 
providing the displayed advertisement, or will be co- 
transmitted subsequently in the signal of the television 
program that the person has selected for viewing. This 
data would then be displayed in a separate window on the 
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mult i -windowed display being used for television viewing. 
An additional set of data could then be obtained by 
clicking on a hypertext link element of the initially 
displayed data. This action would result in downloading 
the desired additional data over the Internet. A soft- 
ware agent installed in this viewing equipment could thus 
log not only what program was being viewed, but could 
also supply additional marketing research data on the 
consumer's use of in-program advertising . 

In addition to providing a device for monitor- 
ing tuning "activity, the software agent of the present 
invention can be used to collect demographic data rela- 
tive to the monitored viewing. In a manual approach to 
the collection of demographic data, the software agent 
could generate a status and prompting display in one 
window of the viewing equipment requiring a viewer or 
user to input the appropriate demographic data. This 
status and prompting display window could have an appear- 
ance similar to that provided by the peoplemeter hardware 
widely used in contemporary audience measurements and 
that displays a status reminder light for each occupant 
of a statistically selected monitoring site. However, in 
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the comparable windowed display arrangement, the name of 
each occupant could be shown along with a status indica- 
tor. 

If the software agent is used with a personal 
computer that can receive digital television programming, 
a variety of more nearly automatic and less intrusive 
approaches (from the television viewer's perspective) 
could be used to collect persons data and update a tiled 
status display. For example, it is well known to iden- 
tify an individual user of a manual input device (such as 
a keyboard -or a mouse) of a computer by the characteris- 
tic rates of data entry and pauses between data entries. 
In a televififion audience measurement, in which the occu- 
pant population to be monitored is very small, measure- 
ments of this sort should be expected to be highly accu- 
rate. Moreover, an increasing number of personal comput- 
ers have video cameras associated with them, so that the 
user of the computer can engage in video conferences by 
telephone. When such equipment is available in a statis- 
tically selected monitoring site, a software agent com- 
prising known head location and face recognition software 
(e.g., as taught by Lu in U.S. Patent No. 4,858,000) can 
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be employed for the identification persons and the col- 
lection of demographic data. 

Certain modifications of the present invention 
have been discussed above. Other modifications will 
occur to those practicing in the art of the present 
invention. For example, the software agents 112, 118, 
and 122, as described above, monitor operating tasks. 
The operating tasks may be operating system events of the 
operating systems used by the equipment in which the 
software agent is resident. Indeed, the software agents 
112, 118, and 122 may monitor any function of the moni- 
tored equipment as long as the desired data is collected. 

Also, a statistically selected monitoring site 
according to each of the embodiments of the invention, as 
described above, includes certain viewing equipment as 
well as certain monitoring equipment. However, it should 
be understood that a statistically selected monitoring 
site according to the present invention may include any 
combination of the equipment incorporated in the embodi- 
ments described above, as well as other and/or different 
equipment . 
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Moreover, the software agent 500 described 
above can be generic so that it can be used wizh any 
piece of equipment with which it is compatible, or it may 
be specialized for each particular apparatus in which it 
is used. Also, the software agent 500 may be arranged to 
detect the all window activities conducted by an audi- 
ence . 

Furthermore, although not shown in Figures 2-5, 
the digital television reception equipment of the embodi- 
ments shown therein may further include auxiliary digital 
television equipment such as a VCR, a digital video disk 
player, a video game, -or other entertainment systems. 

Additionally, as described above, program iden- 
tification codes are detected in order to identify tele- 
vision programs. However, television programs may be 
identified by any indicia such as program names. More- 
over, an interne't page may be identified by its address 
or by an identification code or name or label or, in the 
case of an advertisement on the Internet, by its banner. 
Therefore, any indicia from which a television program or 
content associated with a television program may be iden- 
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tified is referred to herein generically as an identifi- 
cation datum. 

Moreover, the home units 248 and/or 346 may be 
referred to herein an intermediate data collectors. 
5 Accordingly, the description of the present 

invention is to be construed as illustrative only and is 
for the purpose of teaching those skilled in the art the 
best mode of carrying out the invention. The details may 
be varied substantially without departing from the spirit 
10 of the invention, and the exclusive use of all modifica- 

tions which -are within the scope of the appended claims 
is reserved. 
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WE CLAIM: 



1 1. A television audience measurement system 

2 for identifying a viewer selected television program from 

3 among a plurality of television programs simultaneously 

4 broadcast in a broadcast channel, the television audience 

5 measurement system comprising: 

6 a channel detector that detects the broadcast 

7 channel in which the plurality of television programs is 

8 broadcast; and, 

9 acomparitor that sequentially compares an 

10 audio component of the plurality of television programs 

11 broadcast in the detected broadcast channel with an audio 

12 signal of the viewer selected television program so as to 

13 identify the viewer selected television program. 

1 2. The television audience measurement system 

2 of claim 1 wherein the audio signal is detected by a 

3 microphone disposed adjacent a television display. 

1 3 . The television audience measurement system 

2 of claim 1 wherein the audio signal is detected by an 
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1 intrusive connection into equipment related to television 

2 viewing. 

1 4 . The television audience measurement system 

2 of claim 1 further comprising a program identification 

3 datum detector that detects, from the viewer selected 

4 television program, a program identification datum indic- 

5 ative of an identity of the viewer selected television 

6 program . 

1 5. The television audience measurement system 

2 of claim 4 wherein the program identification datum de- 

3 tector comprises a software agent resident in equipment 

4 related to television viewing. 

1 6 . The television audience measurement system 

2 of claim 5 wherein the software agent is arranged to 

3 search for the program identification datum. 

1 7 . The television audience measurement system 

2 of claim 4 wherein the audio signal is detected by a 

3 microphone disposed adjacent a television display. 
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1 8 . The television audience measuremenr sysrerr. 

2 of claim 4 wherein the audio signal is dececred by an 

3 intrusive connection into equipment related zo television 

4 viewing. 

1 9 . The television audience measurement system 

2 of claim 4 wherein the comparitor is arranged to sequen- 

3 tially compare the audio component of the plurality of 

4 television programs broadcast in the detected broadcast 

5 channel w'ith the audio signal of the viewer selected 

6 television program so as to identify the viewer selected 

7 television program if the program identification datum 

8 detector is unable to detect a program identification 

9 datum indicative of an identity of the viewer selected 
10 television program. 

1 10. The television audience measurement system 

2 of claim 1 further comprising a software agent arranged 

3 to detect, from the viewer selected television program, a 

4 data element indicative of an identity of the viewer 

5 selected television program. 
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1 11. The television audience measurement system 

2 of claim 10 wherein the software agent is arranged to 

3 search for a program identification datum indicative of 

4 an identity of the viewer selected television program. 

1 12 . The television audience measurement system 

2 of claim 4 wherein the comparitor is arranged to sequen- 

3 tially compare the audio component of the plurality of 

4 television programs broadcast in the detected broadcast 

5 channel w'ith the audio signal of the viewer selected 

6 television program so as to identify the viewer selected 

7 television program if the program identification datum 

8 detector is unable to detect a program identification 

9 datum indicative of an identity of the viewer selected 

10 television program. 

1 13 . A television audience measurement system 

2 for digital television equipment, wherein the digital 

3 television equipment is disposed in a statistically se- 

4 lected location, the television audience measurement 

5 system comprising: 
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1 a software agent adapted to read, from a daza 

2 packet contained in digital television programming, a 

3 datum identifying a television program, wherein the sofc- 

4 ware agent is stored in memory associated wich the digi- 

5 tal television equipment; 

6 an interface and communication apparatus adapt - 

7 ed to transmit the identification datum to a remotely 

8 located central office. 

1 14 • The television audience measurement system 

2 of claim 13 wherein the digital television equipment 

3 comprises a receiver having a tuner, a microprocessor, 

4 memory, an operating system, and a video display unit. 

1 15. The television audience measurement system 

2 of claim 13 wherein the digital television equipment is a 

3 set top box providing an analog television signal to an 

4 analog receiver. 

1 16 , The television audience measurement system 

2 of claim 13 wherein the digital television equipment 
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3 comprises a set top box providing a digital television 

4 signal to a digital receiver. 

1 17. The television audience measurement system 

2 of claim 13 wherein the digital television equipment 

3 comprises a set top box and a monitor. 

1 18 . The television audience measurement system 

2 of claim 13 wherein the digital television equipment 

3 comprises a personal computer provided with a television 

4 receiver . 

1 19 • The television audience measurement system 

2 of claim 13 wherein the digital television equipment 

3 includes a VCR. 

1 20 o The television audience measurement system 

2 of claim 13 wherein the digital television equipment 

3 includes a digital versatile disk player. 
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1 21. The television audience measurement system 

2 of claim 13 further comprising a person identification 

3 apparatus . 

1 22 . The television audience measurement system 

2 of claim 13 wherein the identification datum is a program 

3 identification code of a television program. 

1 23 . The television audience measurement system 

2 of claim 13 wherein the identification datum comprises a 

3 program name. 

1 24 . The television audience measurement system 

2 of claim 13 wherein the identification datum comprises an 

3 address of an Internet page . 

1 25- The television audience measurement system 

2 of claim 13 wherein the identification datum comprises an 

3 identification code of an Internet page. 
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1 26 . The television audience measurement system 

2 of claim 13 wherein the identification datum comprises a 

3 banner of material viewed by an audience. 

1 27. The television audience measurement system 

2 of claim 13 wherein the identification datum comprises a 

3 signature extracted from a television program viewed on 

4 the digital television equipment. 

1 28. The television audience measurement system 

2 of claim 13 -wherein the software agent is arranged to 

3 detect window activities conducted by an audience. 

1 29. The television audience measurement system 

2 of claim 13 wherein the interface and communication appa- 

3 ratus includes a serial port. 

1 30. The television audience measurement system 

2 of claim 13 wherein the interface and communication appa- 

3 ratus includes a parallel port. 



-68- 



wo 99/59275 



PCTAJS98/14286 



1 31. The television audience measurement system 

2 of claim 13 wherein the interface and communication appa- 

3 ratus includes a universal serial bus. 

1 32. The television audience measurement system 

2 of claim 13 wherein the interface and communication appa- 

3 ratus includes a firewire. 

1 33. The television audience measurement system 

2 of claim 13 wherein the interface and communication appa- 

3 ratus is arranged to send the identification datum to an 

4 Internet service provider via the Internet. 

1 34. The television audience measurement system 

2 of claim 13 wherein the interface and communication appa- 

3 ratus includes an intermediate data collector. 

1 35. The television audience measurement system 

2 of claim 34 wherein the intermediate data collector in- 

3 eludes a store and forward device, and wherein the store 

4 and forward device is arranged to send the identification 

5 datum to the central office via a telephone line. 
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1 36. The television audience measurement system 

2 of claim 34 wherein the intermediate data collector is an 

3 Internet service provider. 

1 37. The television audience measurement system 

2 of claim 34 wherein the intermediate data collector is a 

3 data collection facility located in the central office. 

1 38. The television audience measurement system 

2 of claim 13 wherein the software agent is a software 

3 agent downloaded to the memory associated with the digi- 

4 tal television equipment . 

1 39. The television audience measurement system 

2 of claim 13 wherein the software agent is a plug in soft- 

3 ware agent of the digital television equipment. 

1 40. The television audience measurement system 

2 of claim 13 wherein the software agent is a floppy disk 

3 software agent of the digital television equipment. 
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1 41. A television audience measuremenr sysrerr. 

2 for identifying a viewer selected television prograir. from 

3 among a plurality of television programs broadcast as a 

4 time division multiplexed sequence of data packers in a 

5 broadcast channel, the viewer selected television program 

6 being displayed on a television display in a statisti- 

7 cally selected location, the television audience measure- 

8 ment system comprising: 

9 receiving means for receiving the time division 

10 multiplexed sequence of data packets in the broadcast 

11 channel; 

12 acquiring means for acquiring an audio portion 

13 of the viewer selected television program; 

14 recovering means for recovering audio compo- 

15 nents respectively corresponding to the television pro- 

16 grams contained in the sequence of data packets; and, 

17 comparing means for comparing the audio compo- 

18 nents to the audio portion in order to determine the 

19 viewer selected television program. 
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1 42. The television audience measurement system 

2 of claim 41 wherein the acquiring means comprises a sen- 

3 sor arranged to acquire a representation of a speaker 

4 signal from a speaker associated with the television 

5 display. 

1 43. The television audience measurement system 

2 of claim 41 wherein the acquiring means comprises a con- 

3 nection to audio processing circuitry associated with the 

4 television display. 

1 44 . The television audience measurement system 

2 of claim 41 wherein the receiving means comprises; 

3 an intermediate frequency probe arranged to 

4 acquire an intermediate frequency signal from a viewer 

5 controlled tuner associated with the television display; 

6 and, 

7 demodulating means for demodulating the inter- 

8 mediate frequency signal in order to receive the data 

9 packets. 
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1 45. The celevision audience measurement sysrsm 

2 of claim 41 wherein the receiving means comprises: 

3 a local oscillator frequency probe arranged zc 

4 pick up a local oscillator frequency signal from the 

5 television display; 

6 means for identifying the broadcast channel 

7 from the local oscillator signal; 

8 wherein the recovering means recovers the audio 

9 components from television programs contained in the 

10 identified broadcast channel; and, 

11 wherein the comparing means compares the audio 

12 components to the representation of the speaker signal in 

13 order to determine the viewer selected television pro- 

14 gram . 

1 46. The television audience measurement system 

2 of claim 41 wherein the receiving means comprises a scan- 

3 ning receiver arranged to scan each of a plurality of 

4 broadcast channels and to receive a corresponding plural - 

5 ity of time division multiplexed television programs from 

6 each of the plurality of broadcast channels. 
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1 47. The television audience measurement system 

2 of claim 41 further comprising identifying means for 

3 identifying persons in an audience of the viewer selected 

4 television program. 

1 48. A television audience measurement system 

2 for measuring viewing of a television program viewed on 

3 digital television located in a statistically selected 

4 site comprising: 

5 detecting means for detecting an audio code 

6 embedded In the television program in order to identify 

7 the television program; 

8 extracting means for extracting an audio signa- 

9 ture from the television program in order to identify the 

10 television program; 

11 a software agent arranged to identify the tele- 

12 vision program; and, 

13 selecting means for selecting at least one of 

14 the detecting means, the extracting means, and the soft- 

15 ware agent in order to identify the television program. 
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1 49. The television audience measurement, syscerr. 

2 of claim 48 further comprising retrieving means for re- 

3 trieving an audience measurement data packer from a tele- 

4 vision set in order to identify the television program., 

5 wherein the selecting" means selects at least one of the 

6 retrieving means, the detecting means, the extracting 

7 means, and the software agent in order to identify the 

8 television program. 

1 50. A method of identifying a television pro- 

2 gram selected by a viewer from a set of television pro- 

3 grams broadcast as multiplexed data packets in a viewer 

4 selected broadcast channel, the viewer selected televi- 

5 sion program being displayed on a display portion of an 

6 apparatus tuned to the viewer selected broadcast channel, 

7 the method comprising the steps of: 

8 a) determining the viewer selected broadcast 

9 channel; 

10 b) acquiring an audio portion of the viewer 

11 selected television program; 
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12 c) selecting an audio component associated with 

13 one of the set of television programs broadcast in the 

14 viewer selected broadcast channel; 

15 d) comparing the audio portion with the audio 

16 component in order to determine whether the audio portion 

17 and the audio component match; 

18 e) if the audio portion and the audio component 

19 match, storing a tuning record from at least one of the 

20 audio portion and the audio component; and, 

21 f) if the audio portion and the audio compo- 

22 nent do not match, repeating steps (c) through (f) until 

23 either a match is found or the set of television programs 

24 is exhausted. 

1 51. The method of claim 50 wherein step a) 

2 comprises the step of determining the viewer selected 

3 broadcast channel by use of a channel detector associated 

4 with the apparatus, 

1 52. The method of claim 51 wherein step b) 

2 comprises the step of acquiring the audio portion of the 
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3 viewer selected television program by use of an audio 

4 probe adjacent the apparatus. 

1 53 . The method of claim 52 wherein step c) 

2 comprises the step of selecting the audio component by 

3 use of a digital tuner that is not a portion of the appa- 

4 ratus . 

1 54. The method of claim 50 wherein step b) 

2 comprises the step of accpiiring the audio portion of the 

3 viewer selected television program by use of an audio 

4 probe adjacent the apparatus. 

1 55. The method of claim 54 wherein step c) 

2 comprises the step of selecting the audio component by 

3 use of a digital tuner that is not a portion of the appa- 

4 ratus . 

1 56. The method of claim 50 wherein step c) 

2 comprises the step of selecting the audio component by 

3 use of a digital tuner that is not a portion of the appa- 

4 ratus . 
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1 57. A method of identifying a viewer selected 

2 television program from among a plurality of time over- 

3 lapped television programs broadcast in a viewer selected 

4 broadcast channel, wherein the viewer selected television 

5 program is displayed in a first window of a multi -window 

6 television display, wherein a file is also broadcast in 

7 the viewer selected channel so as to be time overlapped 

8 with the viewer selected television program, wherein the 

9 viewer selected television program and the data file 

10 contain respective labels, wherein material from the file 

11 is displayed in a second window of the multi -window dis- 

12 play, and wherein the method comprises the steps of: 

13 a) reading, from the file, the respective file 

14 label and an identifying datum; 

15 b) finding the television program label asso- 

16 ciated with the respective file label; and, 

17 c) storing a time -stamped record comprising 

18 the identifying datum. 

1 58. The method of claim 57 wherein the file is 

2 a data file. 
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1 59. The method of claim 57 wherein the file is 

2 an image file. 

1 60. The method of claim 57 wherein the file is 

2 logically related to the viewer selected television pro- 

3 gram, wherein the television program label and the file 

4 label are indicative of the logical relation, and wherein 

5 step comprises the step of finding, from the logical 

6 relation, the television program label associated with 

7 the respective data file label. 

1 61. A software agent stored in memory associ- 

2 ated with digital television equipment, wherein the soft- 

3 ware agent is arranged to acquire television audience 

4 measurement data relative to the digital television 

5 equipment, the software agent comprising: 

6 first logging means for logging a television 

7 program identification datum identifying a television 

8 program selected for viewing on the digital television 

9 equipment ; 
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10 second logging means for logging an identifica- 

11 tion datum associated with data corresponding to the 

12 television program selected for viewing on the digital 

13 television equipment; and, 

14 third logging means for logging an Internet 

15 identification datum associated with an Internet task of 

16 the digital television equipment. 

1 62. An apparatus for identifying a viewer 

2 selected television program from among a plurality of 

3 time overlapped television programs broadcast in a viewer 

4 selected broadcast channel and received by digital tele- 

5 vision program reception equipment, wherein the digital 

6 television program reception equipment has a data port, 

7 the apparatus comprising: 

8 reading means connected to the data port for 

. 9 reading program identifying data from among data provided 

10 on the data port; and, 

11 storing means for storing the program identify- 

12 ing data. 
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1 63 . The apparatus of claim 62 wherein the 

2 digital television program reception equipment is a digi- 

3 tal converter. 

1 64. The apparatus of claim 62 wherein the 

2 digital television program reception equipment is a per- 

3 sonal computer . 

1 65. The apparatus of claim 62 wherein the 

2 digital television program reception equipment is a digi- 

3 tal television set. 

1 66. An apparatus for identifying a viewer 

2 selected television program from among a plurality of 

3 time overlapped television programs broadcast in a viewer 

4 selected broadcast channel and received by digital tele- 

5 vision program reception equipment, wherein the digital 

6 television program reception equipment has a data port, 

7 the apparatus comprising: 

8 reading means connected to the data port for 

9 reading program identifying data from among data provided 
10 on the data port; and. 
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11 communicating means for communicating the pro- 

12 gram identifying data to a remote point. 

1 67. The apparatus of claim 66 wherein the 

2 digital television program reception equipment is a digi- 

3 tal converter. 

1 68 . The apparatus of claim 66 wherein the 

2 digital television program reception equipment is a per- 

3 sonal computer. 

1 69. The apparatus of claim 66 wherein the 

2 digital television program reception equipment is a digi- 

3 tal television set . 
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